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We  Rely  on  Software  for  Safe  Aircraft  Operation 


Quantas 

Landing 
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from  Singapore  to 


Even  with  the  autopilot  off,  flight  control  computers  still  '  '  command  control 
surfaces  to  protect  the  aircraft  from  unsafe  conditions  such  as  a  stall/1  the 
investigators  said. 

The  unit  continued  to  send  false  stall  and  speed  warnings  to  the  aircraft's 
primary  computer  and  about  2  minutes  after  the  initial  fault  '  '  generated 
very  high,  random  and  incorrect  values  for  the  aircraft's  angle  of  attack." 

mayday  call  when  it  suddenly  changed  altitude  during  a  flight 
Qantas  said. 


Embedded  software  systems 
introduce  a  new  class  of 
problems  not  addressed  by 
traditional  system  modeling  & 


Liu. 
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causing  the 


Autopilot  Off 

A  "  "  preliminary  analysis"  of  the  Qantas  plunge  showed  the  error  occurred 
in  one  of  the  jet's  three  air  data  inertial  reference  units,  which  caused  the 
autopilot  to  disconnect,  the  ATSB  said  in  a  statement  on  its  Web  site. 

The  crew  flew  the  aircraft  manually  to  the  end  of  the  flight,  except  for  a 
period  of  a  few  seconds,  the  bureau  said. 


jet  to  nosedive. 


vas  cruising  at  37,000  feet  (11,277  meters) 
computer  fed  incorrect  information  to  the  flight  control  system,  the 
Australian  Transport  Safety  Bureau  said  yesterday.  The  aircraft  dropped 
650  feet  within  seconds,  slamming  passengers  and  crew  into  the  cabin 
■~-~iilini~|,  hrfnr--1  p i I r~-g~iin-d  rnni~Tli 


Even  with  the  autopilot  off,  flight  control  computers  still  "  "  command  control 
surfaces  to  protect  the  aircraft  from  unsafe  conditions  such  as  a  stall,"  the 
investigators  said. 

The  unit  continued  to  send  false  stall  and  speed  warnings  to  the  aircraft's 
primary  computer  and  about  2  minutes  after  the  initial  fault  "  "  generated 
very  high,  random  and  incorrect  values  for  the  aircraft's  angle  of  attack. 


> 


< 


"  "This  appears  to  be  a  unique  event,"  the  burea^eaid,  adding  that 


fitted  with  the  same  air-data  computer.  I  he  advisory  is  aimed  at 
minimizing  the  risk  in  the  unlikely  event  of  a  similar  occurrence." 


'ihe  flight  control  computer  then  commanded  a  nose-down  aircraft 

movement,  which  resulted  in  the  aircraft  pitching  down  to  a  maximum  of 
about  8.5  degrees,"  it  said. 

No  "  Similar  Event' 

Airbus  has  advised  that  it  is  not  aware  of  any  similar  event  over  the 
many  years  of  operation  of  the  Airbus,"  the  bureau  added,  saying  it  will 
continue  investigating. 
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Software  Problems 
not  just  in  Aircraft 


ConsumerReports  irg* 


May  7, 2010 

Lexus  GX  460  passes  retest;  Consumer  Reports  lifts  "Don't  Buy" 
label 


Consumer  Reports  is  lifting  the  Dont  Buy:  Safety 
Risk  designation  from  the  2010  Lexus  GX460 
SUV  after  recall  work  corrected  the  problem  it 
displayed  in  one  of  our  emergency  handling  tests. 
(See  the  original  report  and  video:  "Dont  Buy: 
Safety  Risk-2010  Lexus  GX460.") 

We  originally  experienced  the  problem  in  a  test 
that  we  use  to  evaluate  what’s  called  lift-off 
oversteer.  In  this  test,  as  the  vehicle  is  driven 
through  a  turn,  the  driver  quickly  lifts  his  foot  off 
the  accelerator  pedal  to  see  how  the  vehicle 
reacts.  When  we  did  this  with  our  GX460,  its  rear 
end  slid  out  until  the  vehicle  was  almost  sideways. 
Although  the  GX  460  has  electronic  stability 
control,  which  is  designed  to  prevent  a  vehicle 

from  sliding  thp  system masal  ialsamag niiirkK/ 


►  Otfc  101  g, 


This  article  appeared  in 

Many  appliances  now  rely  on  electronic  controls  and  operating  softw  May  2010  Consumer  Reports  Magazine.  _  But  it 

turned  out  to  be  a  problem  for  the  Kenmore  4027  front-loader,  which  scored  near  the  bottom  in  our  February  2010  report. 

Our  tests  found  that  the  rinse  cycles  on  some  models  worked  improperly,  resulting  in  an  unimpressive  cleaning. 


When  Sears,  which  sells  the  washer,  saw  our  February  2010  Ratings  (available  to  subscribers),  it  worked  with  LG,  which  makes 
the  washer,  to  figure  out  what  was  wrong.  They  quickly  determined  that  a  software  problem  was  causing  short  or  missing  rinse 
and  wash  cycles,  affecting  wash  performance.  Sears  and  LG  say  they  have  reprogrammed  the  software  on  the  models  in  their 
warehouses  and  on  about  65  percent  of  the  washers  already  sold,  including  the  ones  we  had  purchased. 

Our  retests  of  the  reprogrammed  Kenmore  4027  found  that  the  cycles  now  worked  properly,  and  the  machine  excelled.  It  now 
tops  our  Ratings  (available  to  subscribers)  of  more  than  50  front-loaders  and  we’ve  made  it  a  CR  Best  Buy. 


If  you  own  the  washer,  or  a  related  model  such  as  the  Kenmore  4044  or  Kenmore  Elite  4051  or  4219,  you  should  get  a  letter  from 
Sears  for  a  free  service  call.  Or  you  can  call  800-733-2299. 


enough  to  stop  the  slide.  We  consider  this  a  safety  risk  because  in  a  real-world  situation  this  could  cause  a  rear 
tire  to  strike  a  curb  or  slide  off  of  the  pavement,  possibly  causing  the  vehicle  to  roll  over.  Tall  vehicles  with  a  high 
center  of  gravity,  such  as  the  GX  460,  heighten  our  concern.  We  are  not  aware,  however,  of  any  reports  of  injury 
related  to  this  problem. 

Lexus  recently  duplicated  the  problem  on  its  own  test  track  and  developed  a  software  upgrade  for  the  vehicle's 
ESC  system  that  would  prevent  the  problem  from  happening.  Dealers  received  the  software  fix  last  week  and 
^egamTotifyin^GX46^wner^^rin^hei^ehicle^r^on^pai^^^^^^^^^^^^^^^^^^^^^^^^ 

We  contacted  the  Lexus  dealership  from  which  we  had  anonymously  bought  the  vehicle  and  made  an 
appointment  to  have  the  recall  work  performed.  The  work  took  about  an  hour  and  a  half. 

Following  that,  we  again  put  the  SUV  through  our  full  series  of  emergency  handling  tests.  This  time,  the  ESC 
system  intervened  earlier  and  its  rear  did  not  slide  out  in  the  lift-off  oversteer  test.  Instead,  the  vehicle 
understeered — or  plowed — when  it  exceeded  its  limits  of  traction,  which  is  a  more  common  result  and  makes  the 
vehicle  more  predictable  and  less  likely  to  roll  over.  Overall,  we  did  not  experience  any  safety  concerns  with  the 
corrected  GX460  in  our  handling  tests. 


How  do  you  upgrade  washing 
machine  software? 
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High  Fault  Leakage  Drives  Major  Increase  in  Rework  Cost 


Aircraft  industry  has  reached  limits  of  affordability 
due  to  exponential  growth  in  SW  size  and  complexity. 


Requirements 

Engineering 


70%  Requirements  & 
system  interaction  errors 


System 

Design 


80%  late  error 
discovery  at  high 
rework  cost 


Acceptance 

Test 


Software 
Architectural 
Design 


System 

Test 


10%,  50.5%  20x 


:or 


Major  cost  savings  through  rework  avoidance 
by  early  discovery  and  correction 

A$10k  architecture  phase  correction  saves  $3M 


Integration 

Test 


Component 

Software 

Design 


Where  faults  are  introduced 

Where  faults  are  found 

The  estimated  nominal  cost  for  fault  removal 

Sources: 

NIST  Planning  report  02-3,  The  Economic  Impacts  of  Inadequate 
Infrastructure  for  Software  Testing,  May  2002. 

D.  Galin,  Software  Quality  Assurance:  From  Theory  to 
Implementation,  Pearson/Addison-Wesley  (2004) 

B.W.  Boehm,  Software  Engineering  Economics,  Prentice  Hall  (1981) 


20%,  16% 
5x 


Unit 

Test 


Total  System  Cost 
Boeing  777  $12B 
Boeing  787  $24B 


Software  as  %  of  total  system  cost 
1997:  45%  2010:  66%  2024:  88% 


Code 

Development 


Post-unit  test  software  rework  cost 
50%  of  total  system  cost  and  growing 
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System  User/Environment 


Mismatched  Assumptions  in  System  Interactions 

System  Engineer  Physical  Plant  Control  Engineer 


Hazards 

Characteristics 

Measurement  Units,  value  range 

Impact  of 

Lag,  proximity 

Boolean/Integer  abstraction 

system  failures 

Air  Canada,  Ariane,  7500  Boolean 
variable  architecture 

Under 

Control 


Operator  Error 

Automation  & 
human  act: 


it 


Data  Stream 
Characteristics 

Latency  jitter  affects 
control  behavior 
Potential  event  loss 
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Compute 
Platform 

Hardware  Distribution  &  Redundancy 

Virtualization,  load  balancing, 
mode  confusion 

Embedded  SW  System  Engine 
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Engineer 


Concurrency 

Communication 

ITunes  crashes  on  dual-cores 


Embedded  software  system 
as  major  source  of  hazards 


Why  do  system  level  failures  still  occur  despite  fault 
tolerance  techniques  being  deployed  in  systems? 
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Model-based  Engineering  Pitfalls 


Inconsistency  between 
independently  developed 
analytical  models 


0  5  Id  ]5 

I  — 1 - I  1 


Confidence  that  model 
reflects  implementation 


The  system 


System  models 


System  implementation 


This  aircraft  industry  experience  has  led  to  the  System 
Architecture  Virtual  Integration  (SAVI)  initiative 
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Why  UML,  SysML  Are  Not  Sufficient 


•  System  engineering 

-  Focus  on  system  architecture  and  operational  environment 
-SysML  developed  to  capture  interactions  with  outside  world,  as  a 

standardized  UML  profile 

-4  pillars/diagrams:  requirements,  parameterics  (added  in  SysML), 
structure,  behavior 

•  Conceptual  architecture 

-  UML-based  component  model 
-Architecture  views  (DoDAF,  IEEE  1471) 

-Platform  Independent  model  (PIM) 

•  Embedded  software  system  engineering 

-OMG  Modeling  and  Analysis  of  Real  Time  Embedded  systems 
(MARTE)  as  UML  profile 

•  Borrowed  Meta  model  concepts  from  AADL 

•  Focus  on  modeling  implementations 

-xUML  insufficient  for  PSM  (Kennedy-Carter,  NATO  ALWI  study) 
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Impact  of  Three  Step  Data  Request  Protocol 
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Operating  as  ARINC653  Partitioned  System 


Data  Consumer  Requirement 

•  Process  data  in  1  second 

Partitions 

•  Provide  space  and  time  boundary  enforcement 

•  Execute  periodically  on  a  static  timeline  at  1  second  rate 
Data  request  protocols  across  partitions 


Requestl  Providel  Request2  Provide2  Request3  Provide3  Process 


|^ConsumerJ  |^ProvideiJ  j^ConsumerJ  |^ProviderJ  |^ConsumeiJ  |^ProvideiJ  j^ConsumerJ 


I 


¥ 


How  much  time  does  consumer  actually  have  to  process  the  data? 
Who  pays  for  the  communication  overhead? 
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Model-based  Engineering  in  Practice 

Modeling  is  used  in  practice 

•  Modeling,  analysis,  and  simulation  in  mechanical,  control,  computer 
hardware  engineering 

Current  practice:  modeling  and  software 

-  Remember  software  through  pictures 

-  MDE  and  MDAwith  UML 
-Automatically  generated  documents 

We  need  language  for  architecture  modeling 

•  Strongly  typed 

•  Well-defined  execution  and  communication  timing  semantics 

•  Systematic  approach  to  dealing  with  exceptional  conditions 

•  Support  for  large-scale  development 
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Challenges  in  Safety-critical  Software-intensive  systems 

An  Architecture-centric  Virtual  Integration  Strategy  with  SAE  AADL 

Improving  the  Quality  of  Requirements 

Architecture  Fault  Modeling  and  Safety 

Incremental  Life-cycle  Assurance  of  Systems 

Summary  and  Conclusion 
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The  Roots  of  AADL 

1990-1998:  MetaH  and  Control-H  by  Steve  Vestal 

•  Strong  typing,  syntax  borrowed  from  Ada 

•  Data  and  event  ports,  Operational  modes 

•  Scheduling  analysis  and  code  generation 

1994:  Application  to  Missile  Guidance  System  by  Vestal  and  Lewis 

•  Three  Week  Port  to  Dual  Processor  Hardware 

1997:  MetaH  Style  for  ACME  by  Peter  Feiler  and  Jun  Li 

•  CMU  ECE  Ph.D.  on  multi-dimensional  analysis  for  Simplex  architectures 
1998:  Error  Model  added  to  MetaH  by  Steve  Vestal 

•  Generation  of  fault  trees  and  Markov  models 
1999:  Requirements  Document  for  AADL  Standard 

•  Industry  input:  packages,  messages 

•  The  best  of  MetaH  and  ACME 
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SAE  Architecture  Analysis  &  Design  Language 
(AADL)  for  Software-reliant  Systems 


The  Mechanical  System 
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AADL  focuses  on  interaction  between  the  three  elements  of  a 
software-reliant  mission  and  safety-critical  systems. 


AADL  and  MBE 
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The  SAE  AADL  Standard  Suite  (AS-5506  series) 

Core  AADL  language  standard  (V2.1-Sep  2012,  Vl-Nov  2004) 

•  Strongly  typed  language  with  well-defined  semantics 

•  Textual  and  graphical  notation 

•  Standardized  XM I  interchange  format 


Standardized  AADL  Extensions 

Error  Model  language  for  safety,  reliability,  security  analysis 
ARINC653  extension  for  partitioned  architectures 
Behavior  Specification  Language  for  modes  and  interaction  behavior 
Data  Modeling  extension  for  interfacing  with  data  models  (UML,  ASN.1,  ...) 


AADL  Annex  Extensions  in  Progress 

Requirements  Definition  and  Assurance  Annex 
Synchronous  System  Specification  Annex 
Hybrid  System  Specification  Annex 
System  Constraint  Specification  Annex 
Network  Specification  Annex 
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System  Level  Fault  Root  Causes 


Violation  of  data  stream  assumptions 


End-to-end  latency  analysis 
Port  connection  consistency 


•  Stream  miss  rates,  Mismatched  data  representation,  Latency  jitter  &  age 


Partitions  as  Isolation  Regions 

•  Space,  time,  and  bandwidth  partitioning 

•  Isolation  not  guaranteed  due  to  undocumented  resource  sharing 

•  fault  containment,  security  levels,  safety  levels,  distribution 


Process  and  virtual  processor  to 
model  partitioned  architectures 


Virtualization  of  time  &  resources 

•  Logical  vs.  physical  redundancy 

•  Time  stamping  of  data  &  asynchronous  systems 

Inconsistent  System  States  &  Interactions 

•  Modal  systems  with  modal  components 

•  Concurrency  &  redundancy  management 

•  Application  level  interaction  protocols 


Virtual  processors  &  buses 
Multiple  time  domains 


Operational  and  failure  modes 
Interaction  behavior  specification 
Dynamic  reconfiguration 
Fault  detection,  isolation,  recovery 


Performance  impedance  mismatches 

•  Processor,  memory  &  network  resources 

•  Compositional  &  replacement  performance  mismatches 

•  Unmanaged  computer  system  resources 


Resource  allocation  & 
deployment  configurations 
Resource  budget  analysis 
&  scheduling  analysis 


Codified  in  Virtual  Upgrade  Validation  method 
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Architecture-Centric  Quality  Attribute  Analysis 


Single  Annotated  Architecture  Model  Addresses 
Impact  Across  Operational  Quality  Attributes 


Safety 
&  Reliabilit 

•MTBF 
•FMEA 

•Hazard 
analysis 


Data 

Quality 

•Data  precision/ 
accuracy 

•Temporal 

correctness 

•Confidence 


Architecture  Mode 


Auto-generated 
analytical  models 

n 


Real-time 

Performance 

•Execution  time/ 
Deadline 

•Deadlock/starvation 
•Latency 


Security 

•Intrusion 

•Integrity 

•Confidentiality 


Resource 

Consumption 

•Bandwidth 

•CPU  time 

•Power 
consumption 
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Multi-Fidelity  End-to-end  Latency  in  Control 
Systems 


Operational 

Environment 


System  Engineer 


System 

Under 

Control 


Control  Engineer 


Control 

System 


Common  latency  data  from  system  engineering 

•  Processing  latency 

•  Sampling  latency 

•  Physical  signal  latency 


S  6 


Impact  of  Scheduler  Choice  on  Controller  Stability 
A.  Cervin,  Lund  U.,  CCACSD  2006 


Unstable  Systems 


-  -  STM 

co_us 

MF 

-I-  h; OFU 


CijS 

'Ati'h 


AADL and  MBE 


Software  Engineering  Institute  Carnegie Mellon  FeMer,oct2o,2oi4 

W  W  O  Of)iA  Parnania  Mallrtn  I 


2014  Carnegie  Mellon  University 


Software-Based  Latency  Contributors 


Execution  time  variation:  algorithm,  use  of  cache 
Processor  speed 
Resource  contention 
Preemption 

Legacy  &  shared  variable  communicatior 
Rate  group  optimization 
Protocol  specific  communication  delay 
Partitioned  architecture 
Migration  of  functionality 
Fault  tolerance  strategy 


Flow  Use  Scenario  through  Subsystem  Architecture 
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Sampling  of  International  Efforts  Leveraging  SAE  AADL 


Compositional 
Timing  Framework 
OSD  2014 


P  Project 
Auto  Code  Gen 
2011-2014 


H 


- V* 

TOPCASED 

Open  Source  Embedded 

Systems  Tool  Framework 

28  partners  €20+M  2005-2009 


OPEES 

Formal  analysis 
2011-2014 


OpenGroup 
Real-Time  Forum 
EU  +  US  partners 
2008-current 


4 -  = 

ra 


D-MILS 

Design  of  Secure  Systems 
2013  -  $4.9M 


ESA  COMPASS 
System  SW  Co-engineering 
2008-current 


ITEA  SPICES 
Model-Driven  Embedded 
Systems  Engineering 
15  partners  €16M  2006-2009 


PM 


1ST  ARTIST 
Embedded  Systems 
Center  of  Excellence 
2007-2012 


AVSI SAVI 

Analysis-based  System  Validation 
12  partners  $20M  2008-current 


EC  ASSERT 
Proof-based  Satellite 
Architectures 
ESA  +  30  partners 
€15M  2004-200™ 


H  pg 

_ MB 


Flex-eWare 
Auto  Code  Generation 
2007-2010 


DARPA  META 
Complex  System 
Engineering 
2010-2012 


ESA  TASTE  ^ 
System  &  SW 
Validation  &  Generation 
2010-current 


PARSEC 

Safety/security 

2010-2013 


r  DARPA  HACMS 

Security  in  CPS 

RC  formal  method 

2013-2015 

AADL  Inspector 
Ellidiss 
2010-current 


PROARTIS 
Partitioned  RT  systems 
2010-2013  €1.8M _ 


RAMSES 

Auto  Code  Generation 
2012-current 


MASIW 

Avionics  Workbench 
2011-current 
$2M  per  year 


Integrated  Clinical 
Environment 
Device  Certification 
FDAKSU 
2011-current 
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Architecture-centric  Virtual  System  Integration 

Evolution,  Maturation  and  Transition 


DARPA 

MetaH 

ACME 


Pilot  Projects  in  US,  Europe,  Japan,  China 


ESA  ASSERT 


JPL 
Mission  Data 
System 


COMPASS 
SE-SW 

C|o -engineering 


System  Architecture  Virtual  Integration  (SAVI)  Software  &  Systems  Engineering 


AADLV1 

Timing 


Software  &  System 
Co-engineering 


Multi-team 

Safety 


Requirements 

Assurance 


SAE  AADL  Standard  &  Tool  Support:  Research  Transition  Platform 


AADLV1 
Error  Model 


OMG 

MARTE 

Embedded 

Systems 


European 

Commission 

SLIM/FIACRE 


DARPA 

META 


US 


& 


ARINC653 

Partitions 


European  Research  Initiatives 


Avionics  Network 
Standards 


System  Safety 
Practice  Standard^ 


DARPA  AADL 
HACMS  Engineering 
ecurity  Workbench 


Regulatory  Guidance 
NRC,  FDA,  UL 


Other  Standards  and  Regulatory  Guidance 


2004 


2008 


2010 


2012 


2014 


2016 


2020 
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Aircraft:  (Tier  0) 


Early  Discovery  and  Incremental  V&V  through 
System  Architecture  Virtual  Integration  (SAVI) 


Aircraft  system:  (Tier  1) 

Engine,  Landing  Gear,  Cockpit, ... 
Weight,  Electrical,  Fuel,  Hydraulics,... 


/d<B«sAccess>> 
ElectricalSupply  { 

HydraulicPower  ^  I  c  (Dun/ 


LRU/IMA  System:  (Tier  2) 

Hardware  platform,  software  partitions 
Power,  MIPS,  RAM  capacity  &  budgets 
End-to-end  flow  latency 


System  &  SW  Engineering: 

Mechatronics:  Actuator  &  Wings 
Safety  Analysis  (FHA,  FMEA) 
Reliability  Analysis  (MTTF) 


J 


— ~P>  FuelSupply 


:<BusAccess>> 


i- 1  Electric  alSupply 

f — HydraulicPower 

Signals 


OEM  &  Subcontractor: 

Subsystem  proposal  validation 
Functional  integration  consistency 
Data  bus  protocol  mappings 


IE 


_ 


Subcontracted  software  subsystem:  (Tier  3) 
Tasks,  periods,  execution  time 
Software  allocation,  schedulability 
Generated  executables 


BusApeg»»p>  analogl _ analog2^j*B»wApcess> >  I  I  li= 


►  navSensorDataFromNSP 
navDataT  oGPAPC 


Repeated  Virtual  Integration  Analyses: 
Power/weight 
MIPS/RAM,  Scheduling 
End-to-end  latency 
Network  bandwidth 


Proof  of  Concept  Demonstration  and  Transition  by  Aerospace  industry  initiative 

•  Architecture-centric  model-based  software  and  system  engineering 

•  Architecture-centric  model-based  acquisition  and  development  process 

•  Multi  notation,  multi  team  model  repository  &  standardized  model  interchange 

■  Multi-tier  system  &  software  architecture  (in  AADL) 

■  Incremental  end-to-end  validation  of  system  properties 
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Multi-Notation  Approach  to  Architecture-centric 
Virtual  System  and  Software  Integration 


AADL and  MBE 


Software  Engineering  Institute  Carnegie Mellon  Feiier,  0*20, 2014 

W  W  O  Of)iA  Parnania  Mallrtn  I 


2014  Carnegie  Mellon  University 


Architecture-centric  Virtual  Integration  Practice 
(AC  VIP) 


Iterative  architecture 
design,  safety  analysis,  and 
requirement  decomposition 


Stakeholder  and 
Quality  Attribute  (QA) 
driven  architecture¬ 
centric  requirement 
specification 


Transformation  and 
code  generation  based 
on  verified  architecture 
specifications 


Architecture-centric  virtual 
integration  and  compositional 
verification  of  requirements 


Testing  against 
verified  specifications 
and  models 


Assurance  plan 
and  execution 
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Outline 


Challenges  in  Safety-critical  Software-intensive  systems 

An  Architecture-centric  Virtual  Integration  Strategy  with  SAE  AADL 

Improving  the  Quality  of  Requirements 

Architecture  Fault  Modeling  and  Safety 

Incremental  Life-cycle  Assurance  of  Systems 

Summary  and  Conclusion 
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Carnegie  Mellon 
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Certification  &  Recertification  Challenges 

Certification:  assure  the  quality  of  the  delivered  system 

•  Sufficient  evidence  that  a  system  implementation  meets  system  requirements 

•  Quality  of  requirements  and  quality  of  evidence  determines  quality  of  system 
Certification  related  rework  cost 

•  Currently  50%  of  total  system  cost  and  growing 
Recertification  Challenge 

•  Desired  cost  of  recertification  in  proportion  to  change 

Improve  quality  of  requirements  and  evidence 

Perform  verification  compositionally 
throughout  the  life  cycle 
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Current  Industry  Practice  in  DO-178B  Compliant 
Requirements  Capture 


Notation 

Enter  an  “s'  in  every  row/column  cell  that  applies 

Sy  stem  Rcqu j  r cm  ent  s 

Data  Interconnect  {ICO} 

c 

§ 

i- 

'  = 

rr 

i> 

OC 

L. 

5fl 

| 

■ 

5 

g 

p 

g 

"3 

p 

0 

% 

s 

0 

13 

> 

p 

-J 

[ 

£ 

0 

J 

H  ar  d war c  Rcq  u  i  r cm  e nts 

English  Test  or  Shall  Statements 

39 

27 

36 

32 

29' 

Tables  and  Diagrams 

31 

30 

30 

19 

18 

UML  Use  Cases 

1 

2 

4 

UML  Sequence  Diagrams 

3 

6 

UML  State  Diagrams 

1 

7 

Executable  Models  (e.g,  Simulink.  SCADE  Suite. 

7 

1 

0 

E 

1 

etc.) 

i 

0 

u 

1 

Data  Flow  Diagrams  (e.g,  Yourdon) 

4 

6 

9 

Need  analyzable  &  executable  specifications 

Other  (Specify)XML 

1 

Operational  models  or  prototypes 

1 

1 

1 

UML 

1 

1 

Tool 

/ 

Enter  an  “x”  m  every  row/column  cell  that 
applies 

System  Requirements 

a 

y 

y 

i 

c 

0 

Q 

High-Level  Software  Requirements 

Low-Level  Software  Requirements 

Hardware  Requirements 

DOORS 

23 

13 

22 

18 

12 

Rational  ROSE* 

1 

3 

RDD-100* 

Requisite  Pro11 

5 

3 

5 

4 

4 

Rhapsody 

1 

SCADE  Suite 

2 

3 

1 

Simulink 

5 

1 

5 

3 

1 

Slate 

1 

1 

1 

Spreadsheet  (e.g..  Microsoft  Excel) 

5 

4 

5 

4 

3 

Statemate 

Word  Processor  (e.g.,  Microsoft  Word) 

19 

20 

18 

17 

16 

YAPS™ 

1 

3 

3 

Designer's  Workbench™ 

1 

1 

Proprietary  Database.  SCADE  like  pic  tool 

1 

1 

Interleaf 

1 

1 

1 

1 

1 

BEACON 

1 

1 

1 

1 

CaliberRM 

1 

1 

1 

1 

1 

XM: 

1 

Wiring  diagram 

1 

1 
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Requirement  Quality  Challenge 


Requirements 

error 

% 

Incomplete 

21% 

Missing 

33% 

Incorrect 

24% 

Ambiguous 

6% 

Inconsistent 

5% 

r  1 

There  is  more  to  requirements  quality  than  “shall”s  and  stakeholder  traceability 
IEEE  830-1998  Recommended  Practice  for  SW  Requirements  Specification 

s 


UserReqts  Technical  Reqts  Design  Test  Cases 


Browsable  links/Coverage  metrics 


IEEE  Std  830-1998  characteristics  of  a  good  requirements  specification: 

•  Correct 

•  Unambiguous 

•  Complete 

•  Consistent 

i  Ranked  for  importance  and/or  stability 

•  Verifiable 

•  Modifiable 

•  Traceable 


System  to  SW  requirements  gap  [Boehm  2006] 

How  do  we  verify  low  level  SW  requirements 
against  system  requirements? 


When  StartUpComplete  is  TRUE  in  both  FADECs  and 
SlowStartupComplete  is  FALSE, 

the  FADECStartupSW  shall  set  SlowStartupInComplete 
to  TRUE 
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Mixture  of  Requirements  &  Architecture  Design  Constraints 

Requirements  and  Design  Information 


Requirements  for  a 
Patient  Therapy  System 

The  patient  shall  never  be  infused 
with  a  single  air  bubble  more  than 
Sml  volume. 

When  a  single  air  bubble  more 
than  Sml  volume  is  detected, 
the  system  shall  stop  infusion 
within  0.2  seconds. 

When  piston  stop  is  received, the 
system  shall  stop  piston  movement 
within  0.0 1  seconds. 

The  system  shall  always 
stop  the  piston  at  the 
bottom  or  top  of  the 
chamber 


The  patient  shall  never  be  infused 
with  a  single  air  bubble  more  than 
Sml  volume. 


2. 


AT  I E  NT  T  H  E  R ARY  SY5T  EM 


INFUSION  SYSTEM 


DRUG 

DELIVERY 

HARDWARE 

AIR  BUBBLE  1 
SENSOR 

PUMP  SYSTEM 

PUMP 

PUMP 

HARDWARE 

CONTROLLER 

When  a  singleair  bubble  more 
than  Sml  volume  is  detected, 
the  system  shall  stop  infusion 
within  0.2  seconds. 


The  system  shall  always 
stop  the  piston  at  the 
bottom  or  top  of  the 
chamber. 


When  piston  stop  is  received, the 
system  shall  stop  piston  movement 
within  0.0  II  seconds. 


Adapted  from  M.  Whalen  presentation 


Typical  requirement  documents  span  multiple  levels  of 

a  system  architecture 

We  have  made  architecture  design  decisions. 

We  have  effectively  specified  a  partial  architecture 
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System  Specification  and  Requirements 
Coverage 


{  Developmental'* 
Requirements 


Quality  attribute  utility  tree 


Modifiability  i 


Assurability  i 


Environmental  Assumptions 


■ 

■ 

■ 

■ 

■ 

■ 

■/ 

^ — 

■ 

■ 

QA  — 

■ 

■ 

■ 

Utility 

■ 

■ 

/ 

Requirements 

Guarantees 

Assumptions 


Environment 


Precondition 

Postcondition 

Invariant 


Interaction  contract: 
match  input  assumption 
with  guarantee 


—  Performance 


' —  Modifiability 


i 

l 


Data 

Latency 


Transaction 
Throughput 
New  products 


(M.M) 

(H.H) 


Deliver  video  in  real  time 


Change 

COTS 


—  Availability 


—  Security 


t 

-c 


COTSSAV 

failures 

Data 

confidentiality 
Data 
integrity 


lucts  I -  Maa 

in  <21 

li”±l  $Ta 

(H.H 

-c 

(H.H 

«T"| 


Add  CORBA  middleware 
20  person-months. 

Change  Web  user  interface 
'  person-weeks 

Power  outage  at  site  1  requires  traffic 
redirected  to  site2  in  <  3  seconds. 

Network  failure  detected  and  recovered 


JH.L) 


Credit  card  transactions  are  secure 
99  909%  ofthe  time 

Customer  D8  authorization  works 
99  999%  ofthe  time 


(  Mission  N 
Requirements  ■ 

. . 1  I 


(  Dependability N 


.  Requirements 

I  r - i 


Function  i 


I 


I 


Behavior  i 

_  _  _  _  _ 

- 1  |  ^ - 1 

Performancei  I  Security  i 


Reliability  i 


r - \ 

Safety  i 


_ /  ' _ * 


Implementation  constraints 


Exceptional  condition 
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Architecture-led  Requirement  &  Hazard  Specification 


System  Specification  Coverage 


Mission-  \ 
critical 

Requiremen  | 
Function 

Behavior 

Performance 

_  .”1 .  / 
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AADL  Error  Model  Scope  and  Purpose 


System  safety  process  uses  many  individual  methods  and  analyses,  e.g. 

•  hazard  analysis 

•  failure  modes  and  effects  analysis 

•  fault  trees  Capture  risk  mitigation  architecture 

•  Markov  processes 


C^stenT^>  Capture  hazards 


(Component)  Capture  FMEA  model 


Goal:  a  general  facility  for  modeling  fault/error/failure  behaviors  that  can  be 
used  for  several  modeling  and  analysis  activities. 

Annotated  architecture  model  permits  checking  for  consistency 
and  completeness  between  these  various  declarations. 


Related  analyses  are  also  useful  for  other  purposes,  e.g. 


•  maintainability 

•  availability 

•  Integrity 

•  Security 


SAE  ARP  4761  Guidelines  and  Methods  for  Conducting  the  Safety 
Assessment  Process  on  Civil  Airborne  Systems  and  Equipment 
Demonstrated  in  SAVI  Wheel  Braking  System  Example 


Error  Model  Annex  can  be  adapted  to  other  ADLs 
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Error  Model  V2:  Abstraction  and  Refinement 

Four  levels  of  abstraction: 

•  Focus  on  fault  interaction  with  other  components 

-  Probabilistic  error  sources,  sinks,  paths  and  transformations 

-  Fault  propagation  and  Transformation  Calculus  (FPTC)  from  York  U. 

•  Focus  on  fault  behavior  of  components 

-  Probabilistic  typed  error  events,  error  states,  propagations 

-  Voting  logic,  error  detection,  recovery,  repair 

•  Focus  on  fault  behavior  in  terms  of  subcomponent  fault  behaviors 

-  Composite  error  behavior  state  logic  maps  states  of  parts  into  (abstracted) 
states  of  composite 

•  Types  of  malfunctions  and  propagations 

-  Common  fault  ontology 


Service  errors 

Omission  Commission 


Omission:  VJ,  (ts,  ^STJ  v  ( \  J  =  -j 


K 


Value  errors  I  Sequence  errors  I 


j  Timing  errors  j  ^  Replication  errors  I 
J  Rate  errors  |  J  Concurrency  errors  I 

I _  _  _  _  _  L  _  —  _  _  _  -i 

Extensions  to  Rowel  LVasi Hades 
Qntokjgiea 


|  Fau  lt  Lattice  far 
Data  streams 


|  Value  errors  |  *  |  Timing  ■ 
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Error  Propagation  Contracts 


Incoming 


NoData 


■5^  Component  C 

ValueError 

NoData 

1  / 

BadValue 

NoResource 


Binding 


BadValue 
NoData 


Outgoing 


LateData 


Legend 


► 


Port 


HW  Binding 


Propagation 
of  Error  Types 
Direction 


Propagated 
Error  Type 


Not 

propagated 


Error  Flow  through  component 
Path  PI. NoData->P2. NoData 
Source  P2.BadData 

Path  processor. NoResource  ->  P2. NoData 


“Not"  on  propagated  indicates  that  this 
error  type  is  intended  to  be  contained. 

This  allows  us  to  determine  whether 
propagation  specification  is  complete. 


Incoming/Assumed 

•  Error  Propagation 

Propagated  errors 

•  Error  Containment: 

Errors  not  propagated 


Outgoing/Contract 

•  Error  Propagation 

•  Error  Containment 


Bound  resources 

•  Error  Propagation 

•  Error  Containment 

•  Propagation  to  resource 
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Original  Preliminary  System  Safety  Analysis 
(PSSA) 


( - ; - - - \ 

System  engineering  activity  with 

focus  on  failing  components. 
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Discovery  of  Unexpected  PSSA  Hazard  through 
Repeated  Virtual  Integration 


system  Ell 
features 

trueairspeed :  out  data  port  DataDictionary :: Velocity; 

flows 

fl:  flow  s 
Latency 
}; 

annex  EMV2  {* 
error  prc 
use  type: 
use  beha\ 
truei 
flows 
efl:errc 
ef2:errc 
properties 
EMV2 : : hazard 
[  crossref€ 
failure  : 
phase  => 
descriptd 
severity 
critical] 
comment  : 


Anticipated: 
No  EGI  data 

V' 


Flight  Mgnt  Syste 


Jcorrupted^ 


system  implem 
subcomponen 

PilotGrip 
Positions 
EGI:  systl  ^ 
FMS:  proces^^ 


Unexpected  propagation  of 
corrupted  Airspeed  data  results 
in  Stall  due  to  miss-correction 


Vibration  causes  boards  to 
touch  which  causes  EGI 
data  corruption 


Actuatorl:  device  Actuator  ; 

Actuator2:  device  Actuator  ; 

FMSProcessor :  processor  PowerPt 
connections 

pilotCmd:  port  PilotGrip. Desin  _ 
sensedPosition :  port  PositionSensor . PositionReading  ->  FMS . Position; 
ActuatorlCmd :  port  FMS.ActCmd  ->  Actuatorl .ActCmd; 

Actuator2Cmd :  port  FMS.ActCmd  ->  Actuator2 .ActCmd; 

vtx :  port  EGI . TrueAirSpeed  - >  FMS. TrueAirSpeed ; _ 

f 


©Outgoing  propagation  {Failure,  CorruptedData}  is  not  handled.  Expected  incoming  {Failure} 


FMS 

Processor 
Operational 
J  Failed 


FMS  Power 


Anticipated: 

NoService 


ctuator 

Cmd 


NoService  | 

””  Stair~”j 


Anticipated:  No 
Stall  Propagation  | 


EGI  maintainer  adds  corrupted  data  hazard  to  model. 
Error  Model  analysis  of  integrated  model  detects 
unhandled  propagation. 


-  s  m.cudiununu - ✓  m.  tuffiui'i .  rs 


Latency  =>  15  ms 
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Recent  Automated  FMEA  Experience 

Failure  Modes  and  Effects  Analyses  are  rigorous  and  comprehensive 
reliability  and  safety  design  evaluations 

•  Required  by  industry  standards  and  Government  policies 

•  When  performed  manually  are  usually  done  once  due  to  cost  and  schedule 

•  If  automated  allows  for 


-  multiple  iterations  from  conceptual  to  detailed  design 

-  Tradeoff  studies  and  evaluation  of  alternatives 

-  Early  identification  of  potential  problems 


*■ - 

ID  Item  Initial  State  Initial  Failure  Mode  1st  Level  Effect  Transition  2nd  Level  Effect:  Transition  3rd  Level  Effect  Severity  M 

1 

Sat Bus 

Working 

Failure 

Failed 

Failed 

Recovery 

Working 

Workir 

1 

Sat Payload 

Working 

Working 

Bus  failure  causes  payload  transition 

Standby 

Standby 

Bus  Recovery  Causes  Payload  Transition 

Workir 

2 

Sat Bus 

Working 

Working 

Working 

5 

2 

Sat_Payload 

Working 

Failure 

Failed 

Recovery 

Working 

5 

Largest  analysis  of  satellite  to  date  consists  of  26,000  failure  modes 

•  Includes  detailed  model  of  satellite  bus 

•  20  states  perform  failure  mode 

•  Longest  failure  mode  sequences  have  25  transitions  (i.e.,  25  effects) 

Myron  Hecht,  Aerospace  Corp. 

Safety  Analysis  for  JPL,  member  of  DO-1 78C  committee 
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Support  of  SAE  ARP4761  System  Safety 
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The  Symptom:  Missed  Stepper  Motor  Steps 

Stepper  motor  (SM)  controls  a  valve 

•  Commanded  to  achieve  a  specified  valve  position 

-  Fixed  position  range  mapped  into  units  of  SM  steps 

•  New  target  positions  can  arrive  at  any  time 

-  SM  immediately  responds  to  the  new  desired  position 

Safety  hazard  due  to  software  design 

•  Execution  time  variation  results  in  missed  steps 

•  Leads  to  misaligned  stepper  motor  position  and  control  system  states 

•  Sensor  feedback  not  granular  enough  to  detect  individual  step  misses 


Software  modeled  and  verified  in  SCADE 

Full  reliance  on  SCADE  of  SM  &  all  functionality 
Problems  with  missing  steps  not  detected 


Software  tests  did  not  discover  the  issue 

Time  sensitive  systems  are  hard  to  test  for. 


Two  Customer  Proposed  Solutions 

Sending  of  data  at  12ms  offset  from  dispatch 
Buffering  of  command  by  SM  interface 
No  analytical  confidence  that  the  problem  will  be  addressed 


Other  Challenge  Problems 

Aircraft  wheel  braking  system 
Engine  control  power  up 
Situational  Awareness  &  health  monitoring 
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Reliability  &  Qualification  Improvement  Strategy 


2010  SEI  Study  for  AMRDEC 
Aviation  Engineering  Directorate 


i 

TIEKT1  [mdTHHKT 


Architecture-led 
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Static  Analysis  & 
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Requirement 

Virtual  System 
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Specification 

Integration 

Verification 

throughout  Life  Cycle 

r 
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Requirements 
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Behavior 

L 

Performance 
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r 

Survivability 

■\ 

Requirements 
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Safety 

Security 
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Model 

Repository 


Component 

Models 


System 

Implementation 


Operational 
&  failure 
modes 
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Contract-based  Compositional  Verification 

Secure  Mathematically-Assured  Composition  of  Control  Models 


Key  Problem 

Many  vulnerabilities  occur  at  component  interfaces. 
How  can  we  use  formal  methods  to  detect  these 
vulnerabilities  and  build  provably  secure  systems? 


TA4  -  Research  Integration  and 
Formal  Methods  Workbench 
Rockwell  Collins  and 
University  of  Minnesota 


ARCHITECTURE-CENTRIC  PROOF 


Formal  System 

L  Requirements 

- 


System  Architects 

System  Design 
Compositional  Verifica 
and  Synthesis 


16  months  into  the  project 

Draper  Labs  could  not  hack  into  the 
system  in  6  weeks 

Had  access  to  source  code 


Technical  Approach 

Develop  a  complete,  formal  architecture  model  for  UAVs  that 
provides  robustness  against  cyber  attack 

Develop  compositional  verification  tools  driven  from  the 
architecture  model  for  combining  formal  evidence  from  multiple 
sources,  components,  and  subsystems 

Develop  synthesis  tools  to  generate  flight  software  for  UAVs 
directly  from  the  architecture  model,  verified  components,  and 
verified  operation  system 


Accomplishments 

•  Created  AADL  model  of  vehicle  hardware  &  software 
architecture 

•  Identified  system-level  requirements  to  be  verified 
based  on  input  from  Red  Team  evaluations 

•  Developed  Resolute  analysis  tool  for  capturing  and 
evaluating  assurance  case  arguments  linked  to 
AADL  model 

•  Developed  example  assurance  cases  for  two 
security  requirements 

•  Developed  synthesis  tool  for  auto-generation  of 
configuration  data  and  glue  code  for  OS  and  platform 
hardware 


Software  Engineering 


Integrated  Approach  to  Requirement  V&V 
through  Assurance  Automation 


]  CastfogtlonCofitrol.a»dl  |  €  requuements.rdaLdiagram 

ASMS  = 


'/  SMS-IA-REQ1:  Input ... 

1/  SMS-0G-REQ2:  Oulp- 
1 1 SMS-POST-REQ3:  At  _ 

'/  SMS-SS-REQ4:  Desif... 

1 1 SMS-SS-REQS:  com_ 

t  SMS-SS-REQ6:  imme_ 
f  SMS-EA-REQ7;  devic... 

■  f  SMS-EA-REQ8:  Powe_ 

■  t  SMS-SS-REQ9:  SMS  h_ 
'  t  SMS-PRE-REQ10:  in  o_ 


t  PCS1AREQ1:  input...  /  PCS-OG  REQ4:  posi_  /  PCS-OG-REQ7: ... 


t  PCS-OC-REQIO;  Co 


fid  ACT 

f  ACI-SS-KEQ1:  com...  /  ACT-IC  REQ3:  daU_.  /  ACT-OG-REQ5: 
f  ACT-SB-REQ2:  que...  /  ACT-SB-REQ4:  Step-  /  ACTSB-REQ6:  _ 

'ACT-IA-REQ7  5tcpc_  ft  AC1 -1A-REQ8:  Stepr...  /  ACT-SS_Rt 

=defivedFrom>> 


/  SMM-1A-REQ1  inp-  t  SMM-OG-REQ2:  ou...  >  SMM-SS-REQ3:... 

f  ecSBy  >  > 

4  SMM-1A-REQ5:  S_ 

/  SMM-SB-REQ4: ... 


t  SMS-REQ3  Subrequifements  are  sufficient 


Requirement  coverage 
Assumption  evidence 


□_  Problems  |  Ud  Properties  [VJ  AAUL  Property  Valuesj  j^j  iraceability  W  Assurance  CTse 


t>  Requirements  Group  SafetyHazards 
d  Q  Requirements  Group  ACT 

t>  t_.  Requirement  ACT-SB-REQ2:  queue  size  zero  and  abor 
[>  /  Requirement  ACT-SS_REQ9:  Homing  command  resul 

t>  /  Req  u  i  rem  ent  ACT-  0  G  -  REQ5 :  M  axStepC  o  u  nt  of  15  i  s  l 
/  Requirement  ACT-SB-REQ6:  StepCount  ==  zero  wher 
/  Requirement  ACT-IA-REQ7:  Stepcount  within  range 
[>  /  Requirement  ACT-SS-REQ1:  command  arrival  driven 

|>  /  Requirement  ACT- SB- REQ4:  StepCount  ==  #  of  step  : 

_ .  _  *  B  _  _  •  ,  ,^-r  T.  r.r. r.  .  '  ft  -  w  - 


Evidence  records  in  terms  of  claims 
that  requirements  have  been  met 


Verified 

Level  [%} 

H 

NaN 

S 

NaN 

S 

100.0 

Q 

NaN 

m 

100.0 

S 

100.0 

Q 

NaN 

H 

100.0 

NaN 

NaN 

0 

100.0 

a 

NaN 

Generated 
assurance  cases 


(J  SafetyHazards 

@  SPI-Reql:  All  safety  h... 

v/  SMS-Haz2:  Sluggish  st.. 

^  SMS-Hazl:  Misaligned  ... 

<<refinedBy>> 

<r<refinedBy>> 

y  Delivery ...  \ 

i  ACT-Haz2:  High  rate  command<arnvalCi^>>^  PCS-Ha^^*^Joes  no... 

<<refinedBy»  \. 

val  time  variation  I... 


lpl... 


I 

f  \£ ACT-Haz2.1:  PCS  executes  U 


<<refinedBy» 

f  SMM-Haz2.2:  Stepper  Motor  is  not 
<  <derivedFrom>  > — 


Safety  hazards  are 
part  of  the  picture 


i/  SMS-SREQ2:  step 


f  SMM-SREQ1:  max(StepDuratio<.verifiqrip^§.^REQ1.  f 
<  <vprifiprlRv>> 


I- 
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Linkage  to  automated  test  harnesses 
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Building  the  Assurance  Case  throughout  the  Life  Cycle 


Continuous  Confidence  Measure  throughout  Life 
Cycle  that  a  System  Meets  its  Requirements 


Architecture-centric  Virtual  Integration 


Architecture  Led 
Requirements  Specification 

V  1 

ts  Deployment 

*  Build 

- s 

Flight  Test 

v  J 

/  Early  Discovery  through  Architecture  Analysis 
System&SW\  leads  to  Assurance  Related  Reworfr** — 1 — ^ - 

\ 

~'<al 


Virtual  Architecture 
Integration  &  Analysis 


Incremental  Evolution  and 
Execution  of  Assurance  Plans 


Incremental  Architecture  & 
Requirement  Evolution 


Requirement 

Coverage 


Design  &  Req 
Refinements 


Verification 


“component 

Software 

Design 


Design  Validation  by 
Virtual  Integration 


Code  Coverage 
Testing 


-Integration 
A  Test 


Code 

evelopment 


\  EwSon nxj  i  yEwienc* 

Auto-generated  Assurance  Cases 


Auto-generation  from 
verified  models 

AADL&SCADE/Simulink 
Ada  SPARK/Ravenscar 
MISRA  C 


Design  &  Req 
Refinement 


Incremental  Contract-based 
Compositional  Verification 
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Build  the  System 
Build  the  Assurance  Case 
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Benefits  of  Architecture-centric  Engineering 


Reduce  risks 

•  Analyze  system  early  and  throughout  life  cycle 

•  Understand  system  wide  impact 

•  Validate  assumptions  across  system 

Increase  confidence 

•  Validate  models  to  complement  integration  testing 

•  Validate  model  assumptions  in  operational  system 

•  Evolve  system  models  in  increasing  fidelity 

Reduce  cost 

•  Fewer  system  integration  problems 

•  Fewer  validation  steps  through  use  of  validated  generators 
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Model-Based 
Engineering 
with  AADL 
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1 5  Years  of  the  SAE  AS-2C  AADL  Committee 

10  Years  since  the  first  publication  of  the  SAE  AADL  standard 

And  many  more  © 


Software  Engineering  Institute  Carnegie Mellon  FeMer,oct2o,2oi4 

- —  ^ ®  OCHA  Pamon ia  Mallnn  I 


AADL  and  MBE 


50 


)2014  Carnegie  Mellon  University 


Contact  Information 


Peter  H.  Feiler 

Principal  Researcher 
RTSS 

Telephone:  +1  412-268-7790 
Email:  phf@sei.cmu.edu 

Web 

Wiki.sei.cmu.edu/aadl 

www.aadl.info 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 

Customer  Relations 

Email:  info@sei.cmu.edu 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 


Software  Engineering  Institute 


Carnegie  Mellon 


AADL and  MBE 
Feiler,  Oct  20,  2014 

©2014  Carnegie  Mellon  University 


